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A3STP ACT 

Program evaluation . is a difficult task, but it must 
be attempted in order to provide tho decision-makers with a basis on 
which to judge the effectiveness of the program and on which to 
reconmend improvements. Realising that difficulties nay exist, such 
as in the unique nature of a specific program or in the lack of 
adequate measurements of ef f ecviveness, it was felt thut a general 
evaluation model should be developed. The Apex model was developed to 
evaluate a program in a rural, economically depressed community in 
North Carolina, with an integrated population of 1,900 students in 
four schools. Grades 1-12. The model was adapted from a previously 
developed model and incorporates process evaluation, product 
evaluation, and a feedback system. It. also makes use of the post-hoo 
method and a control group. Ihxs model was useful in judging the 
effectiveness of the Apex project in relation to specific project 
objectives and in relation to the intent of the legislation which 
funded the project. It is believed that certain aspects of this model 
can be used in developing the needed general model of evaluation. (JS) 
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INTRODUCTION 

The purpose of this paper ia to present, a description of the 
North Carolina exemplary program, henceforth the Apex Program, and the 
evaluation model and methods involved vith the evaluation of this pro- 
gram. While this paper is primarily concerned with the Apex program 
evaluation, many aspects of this approach to evaluation may be general- 
ized to other programs. 



The Apex Program 

The lineage of this exemplary program can be traced directly to 
the general thinking about career development manifested in the legis- 
lation precipitated by the high level of youth unemployment. Under 
Part D (Exemplary Programs and Projects) of the Vocational Education 
Amendments of 1968 (P.L. 90-576, Section 141), Congress defined the 
purpose of exemplary programs and projects: "to stimulate, through 

Federal financial support, new ways to create a bridge between school 
and earning a living for young people, who are (.till in school, who 
have left school either by graduation or dropping out, or who are in 
post-secondary programs of vocational preparation, and to promote co- 
operation between public education and manpower agencies." 

The Apex program, with its three year allocation of approxi- 
mately $400,000 will be funded through the Office of the Commissioner. 
Venn (Policy Paper AVL-V70-1, 1969) pinpointed the priorities that the 
Office of Education had set in light of the 1968 Amendments: 




1. Provisions for broad occupational orientation at the 
elementary and secondary school levels so as to in** 
crease student awareness of the range of options open 
to them in the world of work. 

2. Provision for work experience, cooperative education 
and similar programs, making possible a wide variety 
of offerings in many occupational areas. 

3. Provisions for students not j./eviously enrolled in 
vocational programs to receive specific training in 
job entry skills just prior to the time that they 
leave the school. (Some of these training programs 
might be very intensive and of short duration.) 

•'« . Provision for intensive occupational guidance and 
counseling during the last years of school and for 
initial placement of all students at the completion 
of their schooling. (Placement might be in a job or 
in postsecondary occupational training. Placement 
should be accomplished in cooperation with appropriate 
employment services, manpower agencies, etc.) 

5. Provisions for the grantee or contractor to carry the 
program on with support from regular funding sources 
after the termination of the Federal assistance under 
Part D of P.i. 90-576. (Federal assistance under 
Part 0 cannot exceed three years.) (Policy Paper 
AVL V70-1, 1969). 



This policy statement, together with Section 141 (Vocational 
Education Amendments of 1968, Part D) , guided our efforts in develop- 
ing the exemplary program. 

During the summer of 1969, the Apex community of Wake County, 
North Carolina, was selected as the site for exploratory work in the 
development of a middle grades program by the Center for Occupational 
Education at North Carolina Stats University. This project stimulated 
the interest of school personnel in implementing a total comprehensive 
program in occupational education. The interest displayed by school 
personnel In this area was one of the major fe:tors contributing to the 
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selection of the Apex attendance area as the locale for the present 
project. A number of other factors also were considered during the 
selection process. Apex is the most rural community in Wake County. 

The economic focus of this community is undergoing a rapid transition 
from a predominately agrarian economy toward Increased industrializa- 
tion. Although Apex is located 20 miles from Raleigh, the character of 
the population in the community and problems of providing adequate oc- 
cupational education more closely resemble the typical rural communities 
of North Carolina, and, indeed, of the South, than the larger urban 
areas. The transition period has required a re-examination of the 
needs for occupational education, 'ihe community itself cannot absorb 
the products of the school in its immediate labor force. The socio- 
economic level of the community is relatively low. The per-caplta in- 
come is below the average for Wake County and for North Carolina. The 
Apex attendance area received the largest amount of Title I ESEA funds 
of any school system in the county. Approximately 55 percent of the 
students in the Apex area qualified under Title I support, the highest 
percentage of any Wake County attendance area. Obviously, the project 
focuses on an area that is economically depressed. 

There is an equal distribution of whites and blacks in the Apex 
attendance area. The proportion of black youths in the attendance 
area, 50 percent, is the highest for any attendance area in Wake 
County, and is higher than the proportion of the black population in 
North Carolina. According to Mann (1963) the black population is in- 
creasing proportionally in the target attendance area. The integration 
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plan has been completed for the Apex attendance area. The atudent body 
In each of the schools will be approximately equally divided between 
black and white students. The school dropout rate Is now approximately 
40 percent, and the academic achievement level in the Apex attendance 
area is the lowest of any of the Wake County attendance areas. 

The Apex attendance area Includes four schools: 

1. The Holly Springs Elementary School, which Includes grades 
1-5, with an enrollment of 250 students. 

2. The A. V. Baucum Elementary School, which Includes grades 
1-3, with an enrollment of 250 students. 

3. The Apex Elementary and Junior High School, which includes 
grades 4-8, with an enrollment of 800 students. 

4. The Apex High School, which includes grades 9-12, with an 
enrollment of 600 students. 

The central participants in the project, therefore, are the 1900 
students in the four Apex schools and the 75 administrators and teachers 
who operate the program as well as the parents and other members of the 
community. Since there are no private schools in the Apex attendance 
area, the project will impact upon all youth in the area in grades 1-12. 

Evaluation Specifications 

At the time of formal acceptance of the Apex program, word was 
received that budgetary provisions should he made for an evaluation 
plan to be carried out by a third party. In the case of the Apex pro- 
gram, the Center for Occupational Education is the third party vhich 
has been selected to perform the evaluation (we also are to evaluate 
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Georgia’s exemplary program). The requirements of the evaluation plan 
are: 

An evaluation plan will be carried out by a third par'y 
for evaluating the effectiveness of the program. The plan 
shall describe the steps by which the contractor will: 

A. Determine the extent to which the objectives of the 
program have been accomplished , 

B. Determine what factors either enabled or precluded 
the accomplishment of these objectives, and 

C. Promote the inclusion of the successful aspects of 
the program into vocational education programs supported 
with funds other than those provided under the contract. 
(Department of Health, Education and Welfare, Office of 
Education, Contractu and Grants Division, 1970). 



In the remainder of the piper the evaluation plan will be des- 
cribed and the rationale for selecting this particular plan will be 
discussed. One of our purposes in the presentation of this paper is 
that recently many exemplary programs have been funded and more will be 
funded in the near future. Presumably, all of these programs will re- 
quire evaluation by outside agencies. Although the evaluation system 
that will be presented was developed specifically for the Apex program, 
certain aspects cf the evaluation plan appear to be sufficiently gener- 
aiitable to apply to many other exemplary programs. It is hoped that 
this paper may serve as a stepping stone to the development of a gen- 
eral model of evaluation which may be applied in all exemplary programs. 
The utility of 9uch a model is apparent since it would allow the 
evaluation results of the various programs to be compared, thereby 
greatly reducing the complexity of identifying the factors that con- 
tribute to a successful program, and strengthen the arguments for 
continuation and expansion of the successful aspects cf each program. 




6 



Evaluation of the Project 

The outcomes of establishing this system of evaluation for the project 
are threefold. First, evaluation at the process level allows one to monitor 
the syatcm and its component parts in order to determine if process objec- 
tives are being carried out by project personnel and to identify departures 
from specified procedures. Second, evaluation at the product level enables 
the examination of the results of the project activities in terms of the 
physical entities produced and the behavioral changes produced. Finally, 
evaluation result3 at the product and process level j. ovide the feedback 
information upon which decision-makera can base their system updating decis- 
ions and, given a set of objectives that are fixed for a given time period, 
the set of requirements provided by the U. S. Office of Education can be 
met. A model of such an evaluation system has been developed by Coster and 
Morgan (1969, 1970) and with slight modification can be applied to the 
evaluation of this project. The following section will delineate the 
evaluation model and later the evaluation procedures will be described. 

The Model 

From the twin sources of the individual attributes and the needs 
of society, the mission of vocational education Is specified by legis- 
lation, (Vocational Education Amendments of 1968), albeit somewhat by 
inference. U. S. Office of Education policy papers have produced more 
specific goals for particular programs (Venn, 1969). These must be 
translated into more specific objectives. The specificity and nature 
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of the objectives differ with the level of operation and it is desirable 
to examine a wide range of objectives in order to develop thore objec- 
tives which are cost congruent with the goals of this legislation and 
policy. Once the objectives are specified, the operational procedures 
and resources required to attain the objectives may then be determined. 
The operational procedures and resources constitute the technology of 
education; the combination of human resources, hardware, and software 
which are needed in an appropriate mix to ensure the attainment of th^ 
objectives. Included also in the technology is the know-how by which 
these resources are mixed and applied. The methodology, the emphases, 
the curriculum, and the materials all form part of the technology of 
the educational process. Finally, of concern to project evaluation are 
the actual outputs, or products, of the program. The evaluation model 
to be employed consists of five principal structural components: 

(1) The goals of the program, which are a manifestation of the 
combined mix of the value, structure of society and the attributes of 
the individual are manifested in legislative intent modified or adopted 
in accordance with the State plans and local policies. 

(2) The objectives of the program (desired products). 

(3) The process objectives (desired processes). 

(4) The observed process: 

(a) The operational procedures— i.e. , the methods, 
techniques, emphases, and efforts— being utilised to attain 
the objectives. 

(b) The resources— both materials (including facilities, 
equipment, and materials) and human (including teaching, ad- 
ministrative, supervisory, service and special staff) — provided 
to facilitate, the attainment of the bbjectives. 

(5) The actual outputs or products of the program, as defined 
in terms stated in the product objectives of the project. 
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The static interrelationship of these components is il.lustrated in 
Figure 1. 

Evaluation may be directed toward an appraisal of the processes of 
a project; that is, to an appraisal of the operational procedures and the 
resources available to operate the program and to attain the objectives. 
Evaluation may be directed toward an assessment of the actual outputs or 
products of the program. Traditionally, the major emphasis on evaluation 
has been on the process evaluation regarding such entitles as the 
training and experience of teachers, the hardware and software available 
for the instructional program, the ratio of guidance counselors to 
student enrollment, and the slzo of classrooms and shops while the 
product of educational programs is oftentimes overlooked. 

The assessment of the product of vocational education is more dif- 
ficult to perform. Yet the crux of the evaluation problem is the congru- 
ence between the actual outputs of the program and the product objectives 
of the program. A prime concern of the decision maker is the extent to 
which these two entities are in Juxtaposition. The prime function of an 
evaluation program Is to produce the information necessary to determine the 
extent to which these two entitles are In accord. 

In order for evaluation to be effective, it should be defined in 
terms of information needs of decision makers. Decision makers, therefore, 
were introduced into the model. The complete model is shown in Figure 2. 

The decision makers have been Introduced at two points. First, the decision 
maker (the superintendent) has been introduced between the goals and ob- 
jectives in this model to denote an administrative function. Essentially 
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this illustrates that the decision maker is responsible for specifying 
those objectives congruent with the mission, and harmonious with the goals 
set forth by the legally constivuted bodies. Second, the project director 
has been introduced at a point between the objectives and the process, or 
operational procedures and resources, to denote his implementative function. 
The function of the project director is to design and facilitate the imple- 
mentation of the strategies for the attainment of the objectives of this 
project . 

Thus both decision makers performing administrative and implementa- 
tive functions will be provided with information on both the products, and 
processes of the project. This information will be expressed in terms of 
the degree of attainment of stated objectives. In addition, the evaluative 
function will act as a filter for information obtained about other on-going 
programs that have similar purposes. Other information needs may be ful- 
filled through local inputs within the community, and as needed, evaluators 
may be used to aid in the assessment of the information. 

Evaluation Procedures 

The evaluation will proceed at three levels: (1) the process level, 

(2) the qualitative product level, and (3) the quantitative product level. 

It should be noted that each level of evaluation corresponds to a level of 
the objectives. 

At the process level, evaluation is strikingly similar to a process 
in test construction called "content validity." That is, do experts in the 
area of interest view the project activities as being adequate for achieving 
the stated process objectives. The evaluation function at the process level 
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then is to provide experts with complete and accurate descriptions of pro- 
ject activities that are related to respective process objectives, and to 
catalogue the Judgement of the experts as well as their proposed alterna- 
tives, The results of this phase of evaluation will be reported fully to 
the Superintendent, and to the project director and his staff. Condensed 
versions of the process evaluations and corresponding product evaluations 
will be published annually as Center for Occupational Education Monographs, 
The final evaluation report will be incorporated into the project final 
report and will also be published by the Center for Occupational Education 
in full. Inputs from other exemplary programs will be assessed as needed, 
and reported to the project personnel and Superintendent, 

The product evaluation will consist of comparing the expected re- 
sults, as stated in the product objectives with observed results, The re- 
porting procedures will follow the format described above, The remainder 
of this section will describe how each objective will be assessed, and 
for the sake of brevity, objective numbers that are listed in the product 
objectives subsection will be used instead of quoting the objective. 

The Objectives 

The objectives of the project include both process objectives, which 
refer to programmatic changes and product objectives, which refer to change 
in behaviors of the personnel in the total school system. 

Process Objectives 

The overall process objective of the exemplary program is to imple- 
ment and demonstrate the feasibility of a comprehensive occupational edu- 
cation program in a rural school system which will provide for: 
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1. The Intensification of the counseling-placement function in 
the school system to provide specifically for: 

a. The provision of "realistic information"* about the 
occupational environment to each studen: at a level 
of complexity commensurate with his maturity. 

b. The provision of "realistic information" to each student 
regarding his capabilities and probabilities for success 
(in given occupations). 

c. The provision of practice in decision making to each 
student with emphasis on increasing the student's pro- 
ficiency in making "rational" decisions. 

d. The intensification of individual counseling for students 
immediately prior to leaving school. 

e. The provision of placement services to insure that each 
student who leaves school will be placed in an entry occu- 
pation or in further schooling, and to insure an essential 
continuity between school and community. 

2. The introducation of a program in elementary schools designed 
to provide specifically for: 

a. The integration of occupational Information with basic 
educational skills and the intensification of exposure to 
the range of occupations within the context of the level 
of maturity of the student. 



^Realistic information" refers to the best assessment at the pre- 
sent time, including projections on wage earnings, longevity, and working 
conditions. Of course, there is a possibility that this information may 
not be accurate due to the time lags that exist between the gathering and 
publication of data, and invalidity of our measuring instruments. If our 
instruments were perfect and our projections completely accurate and ra- 
tional decisions always desired, it would not be necessary to burden the 
student with more than a set of optimal occupations for his consideration. 
However, since this is not the case, each individual should be provided 
with as many alternatives as is possible in order to maximize his pro- 
bability of occupational proficiency. 
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b. The development of work habits and realistic attitudes 
toward occupations and work. 

3. The introduction of a program in the junior high school de- 
signed to provide specifically for; 

a. The integration of occupational education with the academic 
curricula at the middle grade level. 

b. Realistic exposure to the range of occupations in the com- 
munity, state, and nation, including up-to-date informa- 
tion as to knowledge, skill and training requirements 

and benefits to be accrued from training. 

c. Realistic exposure to the knowledge of one's self, in- 
cluding the beginning elements of understanding the at- 
tribute mix of the student. 

d. The introduction of the career decision-making process, 
including the choice and consequence of alternatives. 

e. The provision of "hands-on" experience in occupational 
laboratories, and on-site observation of work. 

f. The provision of appropriate skill training for students 
who have decided to leave school prior to completing 
junior high school as a "vestibule function." 

4. The expansion of the occupational education programs in the 
high school program to provide specifically for: 

a. The integration of occupational education with the academic 
program At the secondary level. 

b. The equipping of each secondary school student who does 
not plan to continue formal schooling with a job entry 
skill. 

c. The intensification of the counseling-placement function 
to insure that each student is prepared to obtain employ- 
ment in an occupation. 

d. The expansion of opportunities for cooperative education 
and work study programs. 

e. The provisions for appropriate skill training for students 
who have decided to leave school prior to graduation as a 
"vestibule function." 
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The piocess objectives will be evaluated by expert judgement. The 
processes will be recorded in accordance with these objectives in order 
that other interested parties may evaluate the attainment of the objectives 
These objectives will become more specific as project personnel are em- 
ployed and the program is established. 

Product Objectives 

The product objectives include qualitative and quantitative mani- 
festations of behavior which re expected to change as a result of the 
proposed project. 

Qualitative Objectives 

1. To increase the student’s interest in ami awareness of occupa- 
tions in his community. 

2. To increase the student's interest in academic subject matter- 
areas by incorporating occupational information into the curriculum. 

3. To Increase the interest of parents, teachers, and students 
in occupations . 

4. To increase Interest in employee abilities and attitudes by 
students, teachers, and parents. 

5. To increase satisfaction with curricular offerings. 

6. To increase interest in postsecondary training. 

7. To increase interest in occupational course offerings at the 
high school level. 

8. To increase interest in obtaining entry level skills. 

9. To increase the student’s knowledge of the occupational en- 
vironment and his own abilities. 
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10. To Increase the number of "rational" occupational decisions. 
Quantitative Objectives 

1. To Increase the number of occupations that a student can name 
by 50 percent each year. 

2. To have each student know the occupations of each of the members 
of his Immediate family (mother, father, brothers and sisters). 

3. To Increase the number of "good" work habits that each student 
knows by 50 percent each year. 

4. To increase average academic achievement by 5 centiles each 
year, as measured by the California Achievement Test. 

5. To increase the average daily attendance percentage by 5 per- 
cent each year. 

6. To reduce the number of grade failures by 5 percent per year 
without altering academic standards. 

7. To increase the number of parental conferences requested by 
5 percent per year. 

8. To decrease the dropout rate by 5 percent per year. 

9. To increase the number of students in work experience programs 
by 10 percent per year. 

10. To increase the number of requests for career guidance services 
by 20 percent the first year and 10 percent each of the following years. 

11. To increase to 100 percent in three years, the percentage of 
persons with job entry skills, who do not plan to enter a postsecondary 
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12, To increase the number of students using the occupational in- 
formation center to 80 percent of the students enrolled within three years. 

13, To place all (100 percent) of graduates and dropouts that seek 
employment within three years, 

14, To increase the proportion of students in the high school 
enrolled in vocational programs by 5 percent per year, 

15, To increase the number of course offerings in vocational areas 
by two courses per year, 

16, To increase the number of students applying for postsecondary 
education by 5 percent per year. 

The evaluation system will be employed to monitor and update the 
system objectives and program performance. Since the evaluation is dy- 
namic, the objectives can only be viewed as fixed, prior to the first 
evaluation, which will be based on the performance of the Apex project 
and other exemplary programs, as well as new research findings. 

Qualitative objectives 1, 2, 3, 4, 5, 6, 7, and 8 will be assessed 
by developing questionnaires based psychometrically on Likert scalling 
techniques. The increases in the various qualities will be evaluated by 
a pretest — posttest paradigm, as will all other product objectives. 

Baseline measures will serve as the starting point against which measures 
obtained at a later time will be compared. 

These questionnaires will be designed for persons in various age 
groups that are appropriate for the assessment of given objectives. Con- 
sultants will be employed to aid in the development of these questionnaires 
as well as for other aspects of the product evaluation. 
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Qualitative objective 9 will entail drawing upon items from tests 
that have previously been developed to measure occupational knowledge, 
and design a test appropriate for each age group. The knowledge of in- 
dividual abilities will be assessed by comparing self ratings with best 
results and ratings by other persons. The increase in knowledge would 
correspond to a greater degree of agreement between the self rati”.g and 
the criterion measures. 

The assessment of "rational decisions" (qualitative product ob- 
jective 10) will be limited to grades 7 through 12. The correspondence 
between stated occupation choice, and the probability of success in the 
chosen occupations, will be assessed by staff members. If the probability 
of success is rated below ,5, the occupational choice will be considered 
"irrational decision." 

The quantitative product objectives, like the qualitative product 
objectives, will use baseline measures obtained at the onset of the pro- 
gram as a relative zero point. Quantitative objectives 1, 2, 3, 4, 5, 

6, and 7 will be assessed for grades 1-12, objectives 8, 9, 10, 11, 12, 
and 13 for grades 6-12; and objectives 14, 15, and 16 for grades 9-12. 

Objective 1 will be assessed by simply having each student list 
all of the occupations that he knows and if for example he listed 10 
occupations on the pretest, a 50 percent increase would require that he 
be able to list 15 occupations at the beginning of the second year, 

20 at the beginning of the third year and 25 at the end of the project. 

Objective 2 will be assessed by a listing of occupations of the 
immediate family. The list will be compared with school records. 
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Ojbective 3, like objective one, will be assessed by simply listing 
"good" work habits. The goodness of the work habits that are listed will 
be evaluated by staff members. A simple frequency count of the "good” 
habits will be compared with the baseline measure to ascertain percentage 
increase . 

Objective 4 will be assessed by using population norm deviations 
to obtain centiles at the baseline. These norms will also be used to as- 
certain the centiles from which the baseline measures will be subtracted. 

The average daily attendance at the high school level is approxi- 
mately 83 percent. To fulfill objective 5 the attendance percentage must 
rise to approximately 95 percent. 

Without a change in grade policies, the number of grade failures 
must be reduced by 15 percent of the original number, to fulfill objective 
6. A baseline measure will be used. 

The number of parental requests for consultation about their child's 
career plan must be increased by 5 percent per year in order to fulfill 
objective 7. 

The dropout rate in Apex High and Apex Consolidated is approxi- 
mately 40 percent. To fulfill objective 8 at the end of the program it 
must be 25 percent or less. 

The fulfillment of objective 9 is contingent upon increases in the 
proportion of students in school supported work experience by 15 percent, 
compared to baseline measures. 

The fulfillment of objective 10 is based on increasing requests 
for "career guidance services, 11 as defined by focus of the request, by 
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focus of the request, by 20 per cent over the baseline measure for the 
first year and 20 percent the next two years. 

The fulfillment of objective 11 requires that each person that 
graduates from Apex High School who is not planning to continue education 
will be equipped with entry level Job skills. 

Objective 12 requires that 80 per cent of the students in the middle 
grades must "use 1 ' the Occupational Resources Center. Use is defined as 
spending at least one hour per semester at the Center, 

The fulfillment of objective 13 is contingent upon placing each 
student vh requests placement in a position within a twenty-five mile 
radius of Apex, North Carolina. 

The percentage of students enrolled in vocational programs is 
approximately 25 per cent. To fulfill objective 14, the percentage enrolled 
must increase to 40 per cent. Vocational programs are defined for evalua- 
tion purposes as courses that provide the student with job entry skills. 

Objective 15 Ls fulfilled by adding two courses per year to the 
vocational program curriculum for the three year period. 

To fulfill objective 16 it is necessary to increase the proportion 
of students applying for postsecondary education by 5 per cent per year. 
Post-secondary education means at least one year of education after the 
.completion of high school. 

As was stated before the project process and product objectives will 
be reviewed by the decision-makers constantly. Revisions of the objectives 
are, of course, subject to U.S. Office of Education approval. The 
changes in the objectives may occur on a semi-annual basis the first two 




years and an annual basis thereafter. 



The plan for evaluation in Apex is not based on a random selection 
of students since Apex is an intact sample. Each child will be included 
in the evaluation. A control group composed of randomly selected students 
from schools identified by state personnel as being most advanced in 
vocational eoucation practices and most similar to Apex will be selected. 
These students will only be assessed once at the end of the third year 
of the Apex program. This method is called a post hoc design. 

Before one attempts to develop an evaluation plan, the question 
of why is the evaluation being undertaken must necessarily be answered. 

One answer to this question for the Apex program is assumed to be: 
evaluation is undertaken in order that the decision-maker may be provided 
with information that can be used to improve his program. This function 
corresponds to what Scrlven (1967) calls "formative evaluation". 

Formative evaluation is a developmental form of evaluation in which the 
evaluator's responsibility is providing information upon which Judgements 
about program revisions can be based. Another assumed answer in the Apex 
program is: evaluation is undertaken, in order to determine if a 

program as a whole, is effective. This corresponds to Schiven's definition 
for summative evaluation, Summative evaluation seems to be the type 
of evaluation described in the U. S. Office of Education specifications. 

Continuing on this line of reasoning, certain other assumptions 
have been made. The first assumption is that at best, the Apex program 
is sophisticated guesswork. That is , many aspects of existing research 
and development projects have been combined, into a single program. 

Since the Apex program is unique, one cannot say what effect the inter- 
action of the selected factors wilt have on the program. The second 
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assumption is that even if the Apex proposal were essentially perfect, 
a certain amount of "slippage" would occur because communication systems 
among humans are somewhat less than perfect. Hence i formative evaluation 
will be undertaken, and the program will be revised, or at least reviewed, 
semi-annually for the first two years of operation. 

It is also assumed that the Apex program must justify its effective- 
ness to the funding agency specifically and to the public in general. 

This primarily involves the identification of products that the program 
has produced, as differentiated from those that might be produced without 
the program. K^nce, baseline data and a comparison (control) group 
is necessary. However, the comparison group we have selected would be 
expensive indeed to follow throughout the course of the project, 
therefore we must accept the weaknesses of c post hoc design, with no 
pretest on the control. The differences in the two groups will be 
assumed to be attributable to the program. The summatlve evaluation 
is limited to the third year of the program and objectives will bo 
fixed during this time period. The efficiency of the program canrot 
be determined except by comparison with similar program or indirectly 
by expert opinion. Almost every evaluation plan if not every plan, has 
be?n damned and praised by various sources and for various reasons. 

We expect little better for our plan. For example, Guba (1969) 
virtually annihilated all current models of suruiiatlve evaluation. This 
was followed by Light and Smith (1970) who credited current evaluation 
methods with possessing far greeted power in detecting failure than 
inspiring success. Stufflebeam (1970) attacked summatlve evaluation on 
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similar grounds. Perhaps the most damning of all criticism of $umm9tlve 
evaluation was produced by Wolf (1969) with his tongue-in-cheek "collo- 
qulal method". 

Social psychological research has demonstrated that decisions 
arrived at by a group will achieve greater acceptance than 
decisions arrived at by a individual. This finding is the 
basis of the colloquial method. In applying this method, one 
need merely assemble a group of people who have been associated 
with a particular program to discuss its effectiveness. After a 
brief discussion, the group will usually conclude that the program 
has been indeed successful. This conclusion can then be trans- 
mitted to funding agencies and other school personnel. It is un- 
likely that such evaluations will be challenged since they have 
been arrived at by a group. 

Formative evaluation, on the other hand, has received relatively 
little criticism. This may be due to the fact that the evaluator serves 
in an information-gather ing capacity and does little in the way of 
threatening the existence of the program. The main question raised is: 

Is this really evaluation? 

The Apex Plan combines both types of evaluation and might be 
criticized on the basis that the decision-maker and evaluator lead 
symbiotic existences for such a long period of time as to preclude 
objective summati/e evaluation, yet, realizing this weakness we contend 
that if the evaluator and decision-makers consciously attempt to avoid 
this problem, and since both wish to have the program evaluated objectively; 
the problem will not effect the evaluations objectivity. On the contrary, 
it is contended that through this method the evaluator will be intimately 
familiar with the program, and therefore be in a better position to 
evaluate it objectively. 

Finally, note should be taken that resources were targeted in 
such a manner that each child would be assessed rather than a random 
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sample, thereby precluding a pretest-posttest design. The reasoning 
that went Into this decision was that If the prcgtam was to have maximum 
effect all students’ individual needs must be met. Project personnel, 
in order to meet these needs, must have relevant data on each child, 
rather than the mean of some theoretical population. 

In closing, let me enter a plea that a general model of evaluation 
needs to be developed if we are to approach maximizing the benefits of 
the exemplary programs. For only by direct comparison of elements that 
are common to exemplary programs can the efficiency, and the relative 
effectiveness of the exemplary programs be convincingly conveyed to 
the public. We rope that the model which has been presented here 
might at least serve as a starting point for the development of such a 
general evaluation model. 
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